Development and Validation of a Vocabulary Measure in the Mobile Toolbox.

OBJECTIVE: We describe the development of a new computer adaptive vocabulary test, Mobile Toolbox (MTB) Word Meaning, and validity evidence from 3 studies. METHOD: Word Meaning was designed to be a multiple-choice synonym test optimized for self-administration on a personal smartphone. The items were first calibrated online in a sample of 7,525 participants to create the computer-adaptive test algorithm for the Word Meaning measure within the MTB app. In Study 1, 92 participants self-administered Word Meaning on study-provided smartphones in the lab and were administered external measures by trained examiners. In Study 2, 1,021 participants completed the external measures in the lab and Word Meaning was self-administered remotely on their personal smartphones. In Study 3, 141 participants self-administered Word Meaning remotely twice with a 2-week delay on personal iPhones. RESULTS: The final bank included 1363 items. Internal consistency was adequate to good across samples (ρxx = 0.78 to 0.81, p < .001). Test-retest reliability was good (ICC = 0.65, p < .001), and the mean theta score was not significantly different upon the second administration. Correlations were moderate to large with measures of similar constructs (ρ = 0.67-0.75, p < .001) and non-significant with measures of dissimilar constructs. Scores demonstrated small to moderate correlations with age (ρ = 0.35 to 0.45, p < .001) and education (ρ = 0.26, p < .001). CONCLUSION: The MTB Word Meaning measure demonstrated evidence of reliability and validity in three samples. Further validation studies in clinical samples are necessary.

Remote smartphone monitoring of Parkinson's disease and individual response to therapy.

Remote health assessments that gather real-world data (RWD) outside clinic settings require a clear understanding of appropriate methods for data collection, quality assessment, analysis and interpretation. Here we examine the performance and limitations of smartphones in collecting RWD in the remote mPower observational study of Parkinson's disease (PD). Within the first 6 months of study commencement, 960 participants had enrolled and performed at least five self-administered active PD symptom assessments (speeded tapping, gait/balance, phonation or memory). Task performance, especially speeded tapping, was predictive of self-reported PD status (area under the receiver operating characteristic curve (AUC) = 0.8) and correlated with in-clinic evaluation of disease severity (r = 0.71; P < 1.8 × 10-6) when compared with motor Movement Disorder Society-Unified Parkinson's Disease Rating Scale (MDS-UPDRS). Although remote assessment requires careful consideration for accurate interpretation of RWD, our results support the use of smartphones and wearables in objective and personalized disease assessments.

Smartphone-Based VO2max Measurement With Heart Snapshot in Clinical and Real-world Settings With a Diverse Population: Validation Study.

BACKGROUND: Maximal oxygen consumption (VO2max) is one of the most predictive biometrics for cardiovascular health and overall mortality. However, VO2max is rarely measured in large-scale research studies or routine clinical care because of the high cost, participant burden, and requirement for specialized equipment and staff. OBJECTIVE: To overcome the limitations of clinical VO2max measurement, we aim to develop a digital VO2max estimation protocol that can be self-administered remotely using only the sensors within a smartphone. We also aim to validate this measure within a broadly representative population across a spectrum of smartphone devices. METHODS: Two smartphone-based VO2max estimation protocols were developed: a 12-minute run test (12-MRT) based on distance measured by GPS and a 3-minute step test (3-MST) based on heart rate recovery measured by a camera. In a 101-person cohort, balanced across age deciles and sex, participants completed a gold standard treadmill-based VO2max measurement, two silver standard clinical protocols, and the smartphone-based 12-MRT and 3-MST protocols in the clinic and at home. In a separate 120-participant cohort, the video-based heart rate measurement underlying the 3-MST was measured for accuracy in individuals across the spectrum skin tones while using 8 different smartphones ranging in cost from US $99 to US $999. RESULTS: When compared with gold standard VO2max testing, Lin concordance was pc=0.66 for 12-MRT and pc=0.61 for 3-MST. However, in remote settings, the 12-MRT was significantly less concordant with the gold standard (pc=0.25) compared with the 3-MST (pc=0.61), although both had high test-retest reliability (12-MRT intraclass correlation coefficient=0.88; 3-MST intraclass correlation coefficient=0.86). On the basis of the finding that 3-MST concordance was generalizable to remote settings whereas 12-MRT was not, the video-based heart rate measure within the 3-MST was selected for further investigation. Heart rate measurements in any of the combinations of the six Fitzpatrick skin tones and 8 smartphones resulted in a concordance of pc≥0.81. Performance did not correlate with device cost, with all phones selling under US $200 performing better than pc>0.92. CONCLUSIONS: These findings demonstrate the importance of validating mobile health measures in the real world across a diverse cohort and spectrum of hardware. The 3-MST protocol, termed as heart snapshot, measured VO2max with similar accuracy to supervised in-clinic tests such as the Tecumseh (pc=0.94) protocol, while also generalizing to remote and unsupervised measurements. Heart snapshot measurements demonstrated fidelity across demographic variation in age and sex, across diverse skin pigmentation, and between various iOS and Android phone configurations. This software is freely available for all validation data and analysis code.

Combining accurate tumor genome simulation with crowdsourcing to benchmark somatic structural variant detection.

BACKGROUND: The phenotypes of cancer cells are driven in part by somatic structural variants. Structural variants can initiate tumors, enhance their aggressiveness, and provide unique therapeutic opportunities. Whole-genome sequencing of tumors can allow exhaustive identification of the specific structural variants present in an individual cancer, facilitating both clinical diagnostics and the discovery of novel mutagenic mechanisms. A plethora of somatic structural variant detection algorithms have been created to enable these discoveries; however, there are no systematic benchmarks of them. Rigorous performance evaluation of somatic structural variant detection methods has been challenged by the lack of gold standards, extensive resource requirements, and difficulties arising from the need to share personal genomic information. RESULTS: To facilitate structural variant detection algorithm evaluations, we create a robust simulation framework for somatic structural variants by extending the BAMSurgeon algorithm. We then organize and enable a crowdsourced benchmarking within the ICGC-TCGA DREAM Somatic Mutation Calling Challenge (SMC-DNA). We report here the results of structural variant benchmarking on three different tumors, comprising 204 submissions from 15 teams. In addition to ranking methods, we identify characteristic error profiles of individual algorithms and general trends across them. Surprisingly, we find that ensembles of analysis pipelines do not always outperform the best individual method, indicating a need for new ways to aggregate somatic structural variant detection approaches. CONCLUSIONS: The synthetic tumors and somatic structural variant detection leaderboards remain available as a community benchmarking resource, and BAMSurgeon is available at https://github.com/adamewing/bamsurgeon .

Crowdsourcing biomedical research: leveraging communities as innovation engines.

The generation of large-scale biomedical data is creating unprecedented opportunities for basic and translational science. Typically, the data producers perform initial analyses, but it is very likely that the most informative methods may reside with other groups. Crowdsourcing the analysis of complex and massive data has emerged as a framework to find robust methodologies. When the crowdsourcing is done in the form of collaborative scientific competitions, known as Challenges, the validation of the methods is inherently addressed. Challenges also encourage open innovation, create collaborative communities to solve diverse and important biomedical problems, and foster the creation and dissemination of well-curated data repositories.

The mPower study, Parkinson disease mobile data collected using ResearchKit.

Current measures of health and disease are often insensitive, episodic, and subjective. Further, these measures generally are not designed to provide meaningful feedback to individuals. The impact of high-resolution activity data collected from mobile phones is only beginning to be explored. Here we present data from mPower, a clinical observational study about Parkinson disease conducted purely through an iPhone app interface. The study interrogated aspects of this movement disorder through surveys and frequent sensor-based recordings from participants with and without Parkinson disease. Benefitting from large enrollment and repeated measurements on many individuals, these data may help establish baseline variability of real-world activity measurement collected via mobile phones, and ultimately may lead to quantification of the ebbs-and-flows of Parkinson symptoms. App source code for these data collection modules are available through an open source license for use in studies of other conditions. We hope that releasing data contributed by engaged research participants will seed a new community of analysts working collaboratively on understanding mobile health data to advance human health.

Inferring causal molecular networks: empirical assessment through a community-based effort.

It remains unclear whether causal, rather than merely correlational, relationships in molecular networks can be inferred in complex biological settings. Here we describe the HPN-DREAM network inference challenge, which focused on learning causal influences in signaling networks. We used phosphoprotein data from cancer cell lines as well as in silico data from a nonlinear dynamical model. Using the phosphoprotein data, we scored more than 2,000 networks submitted by challenge participants. The networks spanned 32 biological contexts and were scored in terms of causal validity with respect to unseen interventional data. A number of approaches were effective, and incorporating known biology was generally advantageous. Additional sub-challenges considered time-course prediction and visualization. Our results suggest that learning causal relationships may be feasible in complex settings such as disease states. Furthermore, our scoring approach provides a practical way to empirically assess inferred molecular networks in a causal sense.

Combining tumor genome simulation with crowdsourcing to benchmark somatic single-nucleotide-variant detection.

The detection of somatic mutations from cancer genome sequences is key to understanding the genetic basis of disease progression, patient survival and response to therapy. Benchmarking is needed for tool assessment and improvement but is complicated by a lack of gold standards, by extensive resource requirements and by difficulties in sharing personal genomic information. To resolve these issues, we launched the ICGC-TCGA DREAM Somatic Mutation Calling Challenge, a crowdsourced benchmark of somatic mutation detection algorithms. Here we report the BAMSurgeon tool for simulating cancer genomes and the results of 248 analyses of three in silico tumors created with it. Different algorithms exhibit characteristic error profiles, and, intriguingly, false positives show a trinucleotide profile very similar to one found in human tumors. Although the three simulated tumors differ in sequence contamination (deviation from normal cell sequence) and in subclonality, an ensemble of pipelines outperforms the best individual pipeline in all cases. BAMSurgeon is available at https://github.com/adamewing/bamsurgeon/.

Summary of the DREAM8 Parameter Estimation Challenge: Toward Parameter Identification for Whole-Cell Models.

Whole-cell models that explicitly represent all cellular components at the molecular level have the potential to predict phenotype from genotype. However, even for simple bacteria, whole-cell models will contain thousands of parameters, many of which are poorly characterized or unknown. New algorithms are needed to estimate these parameters and enable researchers to build increasingly comprehensive models. We organized the Dialogue for Reverse Engineering Assessments and Methods (DREAM) 8 Whole-Cell Parameter Estimation Challenge to develop new parameter estimation algorithms for whole-cell models. We asked participants to identify a subset of parameters of a whole-cell model given the model's structure and in silico "experimental" data. Here we describe the challenge, the best performing methods, and new insights into the identifiability of whole-cell models. We also describe several valuable lessons we learned toward improving future challenges. Going forward, we believe that collaborative efforts supported by inexpensive cloud computing have the potential to solve whole-cell model parameter estimation.

clearScience: Infrastructure for Communicating Data-Intensive Science.

Progress in biomedical research requires effective scientific communication to one's peers and to the public. Current research routinely encompasses large datasets and complex analytic processes, and the constraints of traditional journal formats limit useful transmission of these elements. We are constructing a framework through which authors can not only provide the narrative of what was done, but the primary and derivative data, the source code, the compute environment, and web-accessible virtual machines. This infrastructure allows authors to "hand their machine"- prepopulated with libraries, data, and code-to those interested in reviewing or building off of their work. This project, "clearScience," seeks to provide an integrated system that accommodates the ad hoc nature of discovery in the data-intensive sciences and seamless transitions from working to reporting. We demonstrate that rather than merely describing the science being reported, one can deliver the science itself.

Enabling transparent and collaborative computational analysis of 12 tumor types within The Cancer Genome Atlas.

The Cancer Genome Atlas Pan-Cancer Analysis Working Group collaborated on the Synapse software platform to share and evolve data, results and methodologies while performing integrative analysis of molecular profiling data from 12 tumor types. The group's work serves as a pilot case study that provides (i) a template for future large collaborative studies; (ii) a system to support collaborative projects; and (iii) a public resource of highly curated data, results and automated systems for the evaluation of community-developed models.

Systematic analysis of challenge-driven improvements in molecular prognostic models for breast cancer.

Although molecular prognostics in breast cancer are among the most successful examples of translating genomic analysis to clinical applications, optimal approaches to breast cancer clinical risk prediction remain controversial. The Sage Bionetworks-DREAM Breast Cancer Prognosis Challenge (BCC) is a crowdsourced research study for breast cancer prognostic modeling using genome-scale data. The BCC provided a community of data analysts with a common platform for data access and blinded evaluation of model accuracy in predicting breast cancer survival on the basis of gene expression data, copy number data, and clinical covariates. This approach offered the opportunity to assess whether a crowdsourced community Challenge would generate models of breast cancer prognosis commensurate with or exceeding current best-in-class approaches. The BCC comprised multiple rounds of blinded evaluations on held-out portions of data on 1981 patients, resulting in more than 1400 models submitted as open source code. Participants then retrained their models on the full data set of 1981 samples and submitted up to five models for validation in a newly generated data set of 184 breast cancer patients. Analysis of the BCC results suggests that the best-performing modeling strategy outperformed previously reported methods in blinded evaluations; model performance was consistent across several independent evaluations; and aggregating community-developed models achieved performance on par with the best-performing individual models.

Speed of eating as a determinant of bulimic desire to vomit: a controlled study.

Studies of self-induced vomiting of retarded persons have found that the rate of eating and the amount eaten alter this problem. The present study attempted to determine whether this same relation was exhibited by the nonretarded bulimic. A nonretarded bulimic woman provided her subjective ratings of her desire to vomit after eating her taboo foods at a fast versus slowed versus normal rate using a within participant experimental design. The desire to vomit was found to be near absent after the slowed eating of the taboo foods but was at a high level that endured after the rapid or normal eating. These results suggest a neglected determinant of adult bulimia that may be used in clinical treatment, pending confirmation by a larger sample with measures of actual vomiting as was previously evidenced by the inpatient retarded vomiters.

Transient transcapillary exchange of water driven by osmotic forces in the heart.

Osmotic transient responses in organ weight after changes in perfusate osmolarity have implied steric hindrance to small-molecule transcapillary exchange, but tracer methods do not. We obtained osmotic weight transient data in isolated, Ringer-perfused rabbit hearts with NaCl, urea, glucose, sucrose, raffinose, inulin, and albumin and analyzed the data with a new anatomically and physicochemically based model accounting for 1) transendothelial water flux, 2) two sizes of porous passages across the capillary wall, 3) axial intracapillary concentration gradients, and 4) water fluxes between myocytes and interstitium. During steady-state conditions approximately 28% of the transcapillary water flux going to form lymph was through the endothelial cell membranes [capillary hydraulic conductivity (Lp) = 1.8 +/- 0.6 x 10-8 cm. s-1. mmHg-1], presumably mainly through aquaporin channels. The interendothelial clefts (with Lp = 4.4 +/- 1.3 x 10-8 cm. s-1. mmHg-1) account for 67% of the water flux; clefts are so wide (equivalent pore radius was 7 +/- 0.2 nm, covering approximately 0.02% of the capillary surface area) that there is no apparent hindrance for molecules as large as raffinose. Infrequent large pores account for the remaining 5% of the flux. During osmotic transients due to 30 mM increases in concentrations of small solutes, the transendothelial water flux was in the opposite direction and almost 800 times as large and was entirely transendothelial because no solute gradient forms across the pores. During albumin transients, gradients persisted for long times because albumin does not permeate small pores; the water fluxes per milliosmolar osmolarity change were 200 times larger than steady-state water flux. The analysis completely reconciles data from osmotic transient, tracer dilution, and lymph sampling techniques.

An integrative model of coupled water and solute exchange in the heart.

Physiologists have devised many models for interpreting water and solute exchange data in whole organs, but the models have typically neglected key aspects of the underlying physiology to present the simplest possible model for a given experimental situation. We have developed a physiologically realistic model of microcirculatory water and solute exchange and applied it to diverse observations of water and solute exchange in the heart. Model simulations are consistent with the results of osmotic weight transient, tracer indicator dilution, and steady-state lymph sampling experiments. The key model features that permit this unification are the use of an axially distributed blood-tissue exchange region, inclusion of a lymphatic drain in the interstitium, and the independent computation of transcapillary solute and solvent fluxes through three different pathways.